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Abstract, Klein's recognition-primed decision (RPD) framework proposes that experts make decisions by recognizing 
similarities between current decision situations and previous decision experiences. Unfortunately, military personnel are 
often presented with situations that they have not experienced before. Scenario-based training (S8T) can help mitigate 
this gap. However, SBT remains a challenging and inefficient training approach. To address these limitations, the authors 
present an innovative formulation of scenario complexity that contributes to the larger research goal of developing an 
automated scenario generation system. This system will enable trainees to effectively advance through a variety of 
increasing iy complex decision situations and experiences By adapting scenario complexities and automating generation, 
trainees will be provided with a greater variety of appropnately calibrated training events, thus broadening their 
repositories of experience. Preliminary results from empirical testing {N= 24) of the proof -of-concept formula are 
presented, and future avenues of scenario complexity research are also discussed. 


1.0 INTRODUCTION 

Decision-making in the military has evolved 
significantly since the eras of the phalanx and 
the Napoleonic regiment. Their strict, 
hierarchical command and control is 
inappropriate for the asymmetrical conflicts 
warfighters face now. Modern warfighters, 
often acting in small, distributed teams, are 
expected to make numerous, rapid decisions 
based on ambiguous information, all the while 
avoiding conflicts with rules of engagement, 
missions orders, and commander’s intent. 
Unfortunately, lacking personal experience to 
draw upon, junior military personnel are often 
ill-prepared to make such complex decisions, 
and the outcomes of poor decisions may be 
disastrous: incorrect actions engaged, 
unnecessary risks taken, missions 
jeopardized, or casualties received. 

Military personnel faced with unfamiliar 
situations are at a dangerous disadvantage if 
they lack the necessary decision-making 
skills. Unique situations are inherently risky 
and dangerous, fertile ground for poor 
decision-making [1]. Such situations demand 
increased attention, while draining vital 
cognitive resources [2], and they may 
engender anxiety that can detrimentally 
influence decision-making [3], 


However, the more familiar a situation, the 
less risk and danger involved, and the better 
the decision-making. According to Klein’s 
recognition primed decision (RPD) 
framework, decisions are made based upon 
decision-makers' available “pool" of 
internalized previously experienced situations 
(4]. Simply, the more experiences and 
situations individuals can draw from, the more 
likely they are to successfully navigate 
through multiple decision points. In regards 
to simulation, however, more experiences do 
not necessarily translate to a perception of 
scenario fidelity. Multiple experiences within 
simulation that are misaligned to the trainee’s 
level of experience may not be perceived as 
accurately reflecting the complexity to be 
found in actuality. In their experiments. 
Bradley and Shapiro [5] found that at extreme 
levels of complexity, when cognitive capacity 
was taxed, everything became more real to 
participants. The challenge for simulation 
then, is presenting the optimum level of 
complexity to engender a sense of fidelity. 

Just as decision-making in the military has 
evolved, so have its methods of training. 
Today, scenario- based training (SBT), 
defined as the purposeful instantiation of 
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simulated events to create desired 
psychological states [6], is a widely accepted 
instructional approach. One of the strengths 
of SBT is the presentation of varied situations 
that allow trainees to experience real-world 
problems prior to engagement. 

Systematically presenting training along a 
trajectory similar to Bloom’s revised 
taxonomy [7], proceeding from declarative 
knowledge to higher-order levels of 
abstraction and creation, grounds the content 
and delivery in well-documented learning 
theory. Through such training, personnel can 
efficiently and effectively learn to integrate 
multiple skills, cope with realistic distracters, 
practice their higher order cognitive skills, and 
exercise naturalistic decision-making [8], 

1.1 SBT Technologies 

Although SBT can be delivered in a variety of 
ways, the remainder of this paper refers to 
scenario-based military training delivered 
through computer-generated virtual 
environments. 

Currently, in the practice of SBT, scenarios 
are chosen for implementation by a trainer 
based on training objectives, a description of 
the scenario, the trainees’ level of 
experience, and timeline requirements. 
Although training manuals may present 
sequences of training, they typically lack 
explicit recommendations for the content or 
sequencing of training scenarios. Thus, 
without a clear progression of scenarios, 
trainers rely on their own judgment when 
determining the scenarios' tasks and the 
order of presentation of scenarios within a 
set. Consequently, a trainer’s sequencing 
may not align with trainees’ levels of 
experience or performance, and mismatching 
trainees with training events can result in 
diminished— or even negative — decision 
preparedness. 

Recent efforts to develop a SBT system that 
adapts to trainees’ levels of experience have 
been undertaken by the authors, who are 
attempting to create automated methods that 
more effectively advance trainees through a 


variety of increasingly complex training 
scenarios. 

To achieve this goal, the investigators 
needed to operationalize the notion of 
scenario complexity. That is, the authors 
needed to objectively define the subjective 
idea of scenario difficulty. This objective 
formulation is necessary in order to develop 
the software algorithms that perform the 
automated instructional adaptation. 

In the following section the authors present a 
brief definition of scenario complexity, identify 
and describe each of the characteristics used 
in the calculation of scenario complexity, and 
discuss the role of scenario complexity in the 
automation of scenario generation. 

2.0 SCENARIO COMPLEXITY 

To ensure trainees receive scenarios 
appropriate to their experience level, it is 
crucial to objectively define and instantiate 
scenario complexity so that computer-based 
training software can automatically assemble 
appropriate SBT sequences. Successful 
instantiation depends on taking the subjective 
and abstract and making it objective and 
concrete; that is, creating an objective 
computational metric of the subjective notion 
of difficulty. 

The authors define scenario complexity as 
the objective quality of a scenario, which 
interacts with individual characteristics (such 
as trainees’ expertise) to yield an individual’s 
perception of the scenario’s difficulty [9], 

Most importantly, scenario complexity is 
calculated based upon three scenario 
elements that are extrinsic from rather than 
intrinsic to trainees; task complexity, task 
framework and cognitive context moderators. 
To be clear, the authors purposefully refrain 
from attempting to incorporate individual 
perceptions. Subjective interpretation of a 
task’s difficulty or an individual’s affective 
state in relation to a particular characteristic is 
un-actionable and cannot be calculated. It is 
for the purpose of operationalizing and 
Incorporation into program software that 
objective calculation is pursued. For detailed 


921 



description of the formula see Dunne et al. 

[ 10 ], 

2.1 Scenario Complexity 
Characteristics 

The authors’ computational definition of 
scenario complexity is as follows: 

SC=(TC+TF ) * CCM Eq. (1 ) 

where SC = scenario complexity, TC = Task 
complexity, TF = Task framework, and CCM 
- Cognitive context moderators. 

Each of the variables that comprise the total 
SC can be manipulated to increase or 
decrease the total SC. In addition, they can 
be altered to maintain the trainee in the same 
complexity range while at the same time 
presenting variety within the scenario. Each 
variable is calculated through individual 
functions that involve the following sub- 
variables. 

2.1 Task Complexity 

The task complexity component is subdivided 
into component complexity and coordinative 
complexity. 

2.1.1 Component Complexity 

The component complexity characteristic is 
composed of three sub- variables: the 
number of subtasks, required acts, and 
information cues. 

First, component complexity considers the 
number of subtasks. Each training scenario 
is designed around at least one task with 
attendant learning objective(s). A task may 
stand alone, without a sub-task, but 
frequently, tasks include sub-tasks that must 
also be performed. 

Second, each subtask requires one or more 
specific acts. Although a required act is 
principally a pattern of behavior, novice 
trainees internalize the conscious choice to 
engage in the behavior until it becomes 
automatic. Increasing the number of required 
acts presents greater opportunity for 


transitioning conscious behavior to 
unconscious, automatized decisions. 

Finally, each subtask may require monitoring 
of information cues. An information cue is a 
discrete source of information that must be 
monitored and/or processed from the 
environment. The trainee who is aware of 
these cues and chooses to monitor them will 
attain the desired performance more 
efficiently than a trainee who does not. 

2.1.2 Coordinative Complexity 

Coordinate complexity is concerned with the 
integration of subtasks and associated acts, 
which may be necessary for successful task 
completion. These subtasks are integrated 
and involve synchronization of activities to 
achieve the common goal or objective [11]. 

Without coordination of these subtasks and 
acts, trainee performance will suffer. By 
manipulating the degree of integration, 
trainees are presented with increasing levels 
of scenario complexity requiring, in turn, a 
greater number of decision points. 

2.2 Task Framework 

Task framework accounts for the relation 
between task paths and the outcome 
associated with each, and it addresses which 
outcomes are possible in a given task [12]. 

The authors suggest it is the task framework 
characteristic where the interplay of decision 
preparedness, performance and complexity is 
most acute. Tasks such as those with a 
single goal and a single means or path to 
achieve that goal, are well-defined. Tasks 
with multiple goals with several possible 
means or paths to achieve the goals are ill- 
defined tasks. Deciding if a particular means 
or path will achieve the desired goal requires 
a resolution of existing ambiguity; a 
calculation of potentiality for each path’s 
success. Ambiguity and complexity make it 
difficult for decision-makers to determine 
what the possible outcomes might be, let 
alone the value they assign to them [13], 
Increasing ambiguity and complexity is 
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therefore conjectured to be highly influential 
in advancing decision preparedness. 

2.3 Cognitive Context Moderators 

Cognitive context moderators address factors 
that often increase stress and distraction 
present in the scenario and are defined as 
external stimuli that affect the operator by 
increasing load and reducing cognitive 
resources for the task, thus causing less 
complex decisions to appear more complex. 

These moderators can influence the 
resolution or quality of the evidence available 
for supporting judgments. According to 
Macmillan and Creel man [14] when 
attempting a forced-choice judgment to 
identify, for example “known or unknown’’, 
performance will be superior during a clear 
daylight encounter compared to night or 
under hazy conditions because the available 
evidence is superior. 

It must be reiterated that these characteristics 
are built into the scenario; it remains the 
decision of the trainee to engage in these 
acts, monitor cues and satisfy scenario 
criteria. Through presentation of increasing 
levels of complexity, the scenarios contribute 
to the growing pool of experiences from 
which the trainee will draw. 

The authors suggest automated generation of 
scenarios that take trainees on an efficient 
and effective trajectory towards optimized 
decision preparedness can be accomplished 
by this operationalization of scenario 
complexity. To ensure that adaptive 
generation results in positive outcomes, 
systematic implementation of this training 
framework must be grounded in decision- 
making and learning theory. 

The following section describes two major 
theories adaptive scenario generation draws 
upon to increase decision preparedness. 


3.0 OPTIMIZING DECISION 
PREPAREDNESS 

The ability to make timely, appropriate, and 
effective decisions is an essential 
competence for warfighters [1 5], Buch and 
Diehl [15] found that increases in the quality 
of decision-making have largely been by- 
products of in-field experience. However, 
they concluded that judgment and decision- 
making capability can be improved through 
training, incorporating situation-specific 
exercises and increasing the variety of 
variables as training progressed. 

With an objective value for each scenario's 
complexity, variables can be manipulated to 
increase or decrease variety and complexity. 
This manipulation must be calibrated to 
trainee performance, aligning the level of 
complexity associated with a scenario to the 
instructional needs of trainees. However, in 
order to align the scenario to an individual’s 
training needs— making the scenario that is 
"just right” —the simulation must employ a 
systematic instructional methodology [16], 

The following section describes how the 
instructional methodology, based on Bloom's 
taxonomy, is supported through the 
operationalization of scenario complexity for 
SBT. Also discussed is the role of automated 
scenario generation and how it is utilizes 
Klein’s recognition-primed decision-making 
framework and, by extension, Klein and 
Baxter’s cognitive transformation theory [17] 
to improve decision preparedness. The 
section also describes the well-documented 
instructional efficacy of Vygotsky’s zone of 
proximal development [16], used to enable 
proper alignment and sequencing of 
scenarios. 

3.1 Recognition-primed Decision- 
making (RPD) 

Recent decision-making theories have 
focused on decisions that are made in 
complex situations with high stakes, 
uncertainty and time pressure [13], 
Naturalistic decision-making theory (NDM) 
attempts to explain such decisions by 
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suggesting that a decision-maker continually 
assesses a situation in order to recognize 
familiar characteristics and make judgments 

[19] , NDM has been observed across 
domains, such as firefighting and the military 

[20] , where the decision-maker initially 
assesses the situation, looks for familiar 
patterns or prototypes, determines which 
goals make sense, identifies the relevant 
cues to expect, and determines what action 
should will be most appropriate. 

Under the umbrella of NDM research, the 
RPD model has gained significant influence 
over the past 10 to 15 years. This model is 
based on the supposition that, in complex 
situations, humans usually make decisions 
based on the recognition of similarities 
between the current decision situation and 
previous decision experiences. Simply, Klein 
proposes that decisions are made based on 
the recall of the consequences of previous 
decisions made in similar situations. 

Cognitive studies have shown that over 95% 
of human decisions conform to the RPD 
model in time-stressed situations [21] — the 
very type of situation frequently encountered 
by military personnel. 

An extension of RPD is cognitive 
transformation theory [CTA], This theory 
states that as novices progress towards 
expertise they develop "knowledge shields” 
that serve to protect their established 
concepts. These shields can negatively 
affect knowledge and skills acquisition and, in 
the operational theater, may lead to situations 
where default decisions are made based on 
biased judgment rather than experience [17], 

However, Klein and Baxter reference Waller, 
Hunt, and Knapp [22] who found that varied 
and sufficient exposure to virtual training 
environments provides trainees with the 
needed practice to construct valid mental 
models and alleviate the obstacles created by 
such knowledge shields [17]. 


3.2 Zone of Proximal Development 
(ZPD) 

The theory of the zone of proximal 
development (ZPD) is the scientific basis for 
why training (in this case, scenario) difficulty 
must be appropriately matched to trainees’ 
current skill levels. The ZPD Is the range 
within which learning and training tasks are 
neither too hard nor too easy. According to 
Vygotsky, development will only occur when 
a trainee is confronted by a task that lies 
within the zone, because if a task is too easy 
then no development will happen (although 
gains in fluency and accuracy may occur 
simply through repetition) [23] and if a task is 
too difficult to complete successfully, no 
cognitive development will occur [24] and 
motivation may suffer. 

3.3 Implementation 

For illustrative purposes let us say during 
SBT, a trainee performs exceptionally well. 
They have attained the goals and 
successfully navigated decision points. 
Another trainee has not performed so well. 
They have not attained the goals, and their 
decisions were inadequate. Following the 
principle of ZPD the next scenario in 
sequence should be neither too difficult nor 
too easy. Due to such differing performance, 
under today’s SBT approach, it cannot be 
expected that one scenario will address the 
learning needs of both trainees. However, if 
the next scenario in sequence could increase 
in complexity for the high performer while 
either maintaining the current complexity 
level, with variation, or remediating at a lower 
complexity range for the other trainee, then 
the training should be more effective and 
efficient. Achieving this is the authors’ goal. 

Scenario complexity ranges derived from the 
computational formula developed by the 
authors, allow software to determine, with or 
without manual input from the trainer, the 
appropriate sequence of experiences for both 
of these trainees. If, according to ZPD, 
negative training exists outside both the 
upper and lower bounds, then scenario 
complexity defines the upper and lower 
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ranges of the complexity of the appropriate 
sequenced scenario. 

Aligning scenario sequencing to trainee 
performance, therefore, ensures that trainees 
are not presented with decision points and 
situations which are too hard or too easy, but 
remain in the area of ideal learning, while 
presenting varied situations and experiences 
increasing decision preparedness. 

To establish empirical basis for this theory of 
scenario complexity an initial study is being 
conducted. The focus of this study, as well 
as its design, methodology, and preliminary 
findings are presented in the following 
section. 

4.0 EXPERIMENTATION 

A pilot study was conducted to ascertain if the 
researchers’ theory, equations, and 
hypotheses are usable and conceptually 
sound. Incorporating lessons learned from 
this pilot study, any reformulation to the 
original equation that is suggested by the 
results will precede further research. The 
end-state study will include construction of 
dynamic scenario- based training of varying 
levels of complexity designed by military 
personnel. 

4.1 Design and Methodology 

This pilot study involved a single sample 
group {N= 24) comprised of undergraduate 
students recruited from a large southeastern 
university. Each participant completed four 
surveys, which asked participants to indicate 
their perception of the complexity levels of 
various situations. These situations’ 
complexity levels were calculated a priori by 
the authors, using the described scenario 
complexity formula. 

The first series of 6 questions asked the 
participants to indicate on a scale of 1-20 how 
simple or complex they thought it would be to 
drive a car: (Qi) in an empty parking lot, (Q 2 ) 
to a familiar destination in light traffic, (Q 3 ) to 
an unfamiliar destination in a familiar city, 

(Q 4 ) to an unfamiliar destination in an 
unfamiliar city using a map, (Q 5 ) to a familiar 


destination in heavy traffic and severe 
thunderstorm, and (Q s ) to an unfamiliar 
destination using a map, in light traffic and 
mild rain. 

The second series of 6 questions asked the 
participants to indicate how simple or 
complex they thought it would be to: juggle 
three balls of different sizes (Q 7 ), juggle two 
balls and walk at the same time (Q s ), juggle 
two balls (Q 9 ), juggle three balls of the same 
size and walk (Qio), juggle three balls of the 
same size while reciting the ABC’s (Q11), and 
juggle two balls of different sizes and walk 
while reciting the ABC’s (Q 12 ). 

The third series of 6 questions asked the 
participants to indicate how simple or 
complex they thought it would be to drive a 
car: in an empty parking tot while talking on 
the phone (Q 13 ), to a familiar destination in 
light traffic while talking on the phone (Q 14 ), to 
an unfamiliar destination while talking on the 
phone (Qib), to an unfamiliar destination with 
a map while talking on the phone (Q 16 ), to a 
familiar destination in heavy traffic and severe 
thunderstorm while talking on the phone (Q 17 ) 
and, to an unfamiliar destination with a map 
in light traffic and mild raid while talking on 
the phone (G 18 ). 

The fourth series of questions set the 
participants into a single scenario comprised 
of 7 different situations and were asked to 
indicate how simple or complex they thought 
each situation would be to: drive to their 
friends house {Q 19 ), drive to their friends new 
house where they've never been before {G 20 ), 
surf (Q 2 i), play a ping-pong toss game (Q 22 ), 
play the toss game with a fan aimed out over 
the game {Q 23 }, choose from a large menu of 
pizza toppings under a time constraint (Q 24 ) 
and to pick one of two sodas (Q25). 

4.2 Hypotheses 

The authors hypothesized that the scenario 
complexity computation calculated by the 
authors would be similar to the results 
identified by the participants, in other words, 
the authors expected that the formula would 
yield relatively comparable results, regardless 
of who assessed the described scenarios. 
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4.3 Results 

An independent one-sample Mest was 
conducted on the participant (n=24) 
responses to evaluate whether their means 
were significantly different from the test 
values established a priori. 


In the graph below the a priori complexity 
levels and participants’ responses are 
illustrated. The light line represents the 
calculated levels while the dark line 
represents the mean levels of the aggregated 
participant responses. 


Z-scores were also derived to ascertain the 
degrees of agreement among participants’ 
values. Z-scores between -1.249 and 1.249 
indicate relative agreement as to a situation’s 
difficulty. Z-scores < -1.25 or > 1.25 
represent significant disagreement. 

The following table shows the z-scores for 
each item. The total mean and standard 
deviation are noted below. 


Calculated Z-scores* 
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Table 1: Calculated Z-scores, means, and Mest 


results; *N= 24, 95% confidence. 


Expected levels and participant response 
means 



Figure 1 : Expected and participant complexity 
levels 


4.4 Conclusion 

Two salient conclusions can be drawn from 
the results of this study. First, Z-scores 
indicate significant disagreement at both ends 
of the complexity spectrum, suggesting that 
the low- and high-end complexity scenarios 
have a wider range of responses than do the 
middle-ground values. Second, there was a 
significant level of agreement of perceived 
complexity relating to those situations which 
occupy the middle-ground. 

5.0 DISCUSSION 

These results indicate areas of both promise 
and challenge. As a proof of concept, the 
authors believe their formalization of scenario 
complexity is headed in the correct direction; 
however, there are adjustments and 
considerations which must be accounted for 
in future iterations. Attempting to quantify 
subjective perceptions is rife with 
inconsistency. Two participants may give two 
different values for the same situation even 
though they both perceive the situation as 
being very simple. Further, levels of 
experience may have, as Klein suggests, 
played a discriminating role. Participants with 
a larger repository of experiences to draw 
from may have perceived the described 
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situations with a lower level of difficulty 
compared to those with fewer experiences. 
This argues not only for the support of RPD 
but also for the necessity to adapt training 
and complexity levels to the trainee within 
their zone of proximal development or risk de- 
motivation and inefficient training. 

Additionally, disparity between participant 
responses and the a priori calculated values 
may point to participants’ over-confidence 
and/or a lack of understanding of the 
characteristics of the situation That is, while 
the scenario complexity formula takes into 
account such characteristics as number of 
task outcomes and cognitive context 
moderators, participants may not be able to 
readily identify such attributes. Similarly, 
participants may identify such factors, but 
may not consider them impactful. For 
instance, in the item that asked participants 
for an evaluation of their ability to utilize a 
cell-phone while in traffic during a thunder 
storm, over-confidence may lead some to 
inaccurately assess the challenge of 
performing under such conditions. 

Lessons from this pilot study suggest further 
refinement of both the formulation of scenario 
complexity as proposed, and of the research 
design. In order to investigate validity and 
reliability across the entire spectrum of 
complexity, future research requires 
recalculation of the weight given to each 
characteristic, and reformulation of the task- 
framework equation in particular, in addition 
to controlling for levels of experience and 
including more detailed instructions for 
participants. 

Branching outward from this avenue of 
investigation, promising areas of research 
include, but are certainly not limited to, 
investigating the role of cognitive load on 
scenario complexity. That is, does objective 
calculation of complexity adequately address 
both a novice and expert trainee’s different 
requirements of cognitive load? Second, in 
respect to integration of team members, how 
do multiple agents impact an individual’s 
performance in dynamic, adaptive scenarios? 


Third, to what degree is perception of 
scenario fidelity affected by increasing 
scenario complexity following Vygotsky’s 
ZPD? Finally, what role do personal 
characteristics such as metacognition and 
self-efficacy play in trainees’ performance? 
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